System and apparatus for improved eye tracking using a mobile device

ABSTRACT

Embodiments are disclosed for using an apparatus to perform pupil identification. An example method includes measuring, by a light sensor of the apparatus, an ambient infrared light level. The example method further includes determining whether the measured ambient infrared light level exceeds a threshold level. The example method further includes performing pupil identification on a user based on the determination of whether the measured ambient infrared light level exceeds the threshold level. In this regard, the example method involves performing bright pupil identification in an instance in which the measured ambient infrared light level exceeds the threshold level, and performing dark pupil identification if not. Corresponding apparatuses and non-transitory computer readable storage media are also provided.

TECHNOLOGICAL FIELD

Example embodiments of the present invention relate generally to eyetracking and, more particularly, to eye tracking using a mobile device.

BACKGROUND

The inventor has discovered problems with existing mechanisms for eyetracking. Through applied effort, ingenuity, and innovation, theinventor has solved many of these identified problems by developingsolutions embodied by the present invention and described in detailbelow.

BRIEF SUMMARY

Eye tracking systems have many useful applications in a wide array ofindustries. As one example, eye tracking systems can be used as activetools in new types of human-machine interfaces responsive to human eyemovement. As another example, eye tracking systems can be used passivelyto enhance the understanding of human behavior in various situationswhere gaze analysis can provide insight into human thought processes.Eye tracking technology thus has potential to modify and/or inform howwe perform a wide range of activities in the future. In this regard, thebenefits of eye tracking technology are likely to be particularlyvaluable when the tracking can be performed by a mobile device, whichcan enable eye tracking to be performed in changing real-worldenvironments.

However, the inventor has identified problems with existing eye trackingsystems that prevent the wider utilization of existing eye trackingsystems. First, existing eye tracking systems do not intelligentlyutilize both dark pupil identification and bright pupil identificationmethods. These identification methods are effective in differentenvironmental conditions, and thus the inventor has determined that theeffectiveness of eye tracking systems can be improved by intelligentlytoggling between these underlying pupil identification methods. Somesystems employ only a single pupil identification method, but not two.Other systems employ multiple pupil identification methods, but do so inall scenarios, or select from among different pupil identificationmethods on an ad hoc basis or in response to performance of complexevaluations of the quality of the output of each pupil identificationmethod. A need therefore exists for a system that efficiently togglesbetween dark pupil identification and bright pupil identificationprocesses to maximize eye tracking accuracy while reducing the resourcecost of deciding when to switch identification methods.

Moreover, existing eye tracking systems do not employ smartphones ortablet devices. Instead, existing eye tracking systems typicallycomprise peripheral devices that connect to a desktop or laptopcomputer, or comprise special-purpose eyewear (e.g., virtual realityheadsets or specially designed glasses). But reliance on peripheraldevices prevents mobility of an eye tracking system, and thus foreclosesmany potentially useful applications of eye tracking technology. Andwearable technology alienates a large segment of the consumer base thatis be unwilling to purchase yet another type of device. Accordingly, aneed exists for an eye tracking system that improves the likelihood ofconsumer adoption of eye tracking technology.

Finally, existing eye tracking systems require special-purpose equipmentto perform both dark pupil identification and bright pupilidentification. But regardless of how effective an eye tracking systemmight be, it cannot be realistically deployed to the mass market unlessthe equipment it relies upon itself is commonly available. Accordingly,a need exists for an eye tracking system that functions using mobiledevice technology that is both available and ubiquitous today.

Example embodiments described herein provide solutions to theseproblems, as well as others. In one example embodiment, a method isprovided for an eye tracking system to efficiently toggle between pupilidentification methods. In another example embodiment not mutuallyexclusive of the first, an apparatus is configured for pupilidentification while also having a form factor and functionality similarto that of the smartphone or tablet devices currently utilized byconsumers today. And in yet another example embodiment not mutuallyexclusive of the first or second, an apparatus for pupil identificationmay comprise an existing smartphone or tablet device that is modified bysoftware to provide eye tracking functionality that historically hasrequired special-purpose equipment.

The above summary is provided merely for purposes of summarizing someexample embodiments to provide a basic understanding of some aspects ofthe invention. Accordingly, it will be appreciated that theabove-described embodiments are merely examples and should not beconstrued to narrow the scope or spirit of the invention in any way. Itwill be appreciated that the scope of the invention encompasses manypotential embodiments in addition to those here summarized, some ofwhich will be further described below.

BRIEF DESCRIPTION OF THE DRAWINGS

Having described certain example embodiments of the present disclosurein general terms above, reference will now be made to the accompanyingdrawings, which are not necessarily drawn to scale.

FIGS. 1A and 1B comprise respective photographs of pupils undergoingbright pupil identification and dark pupil identification processes,respectively.

FIG. 2 illustrates a schematic diagram of the components of an exampleapparatus that may perform operations described herein, in accordancewith some example embodiments described herein.

FIG. 3 illustrates a schematic block diagram of an example mobile devicethat may perform certain operations described herein.

FIG. 4 illustrates a schematic block diagram of another example mobiledevice that may perform certain operations described herein.

FIG. 5 illustrates a flowchart describing example operations performedby an apparatus in accordance with some example embodiments describedherein.

FIG. 6 illustrates an example user interface that may be presented to auser in accordance with some example embodiments described herein.

DETAILED DESCRIPTION

Some embodiments of the present invention will now be described morefully hereinafter with reference to the accompanying drawings, in whichsome, but not all embodiments of the inventions are shown. Indeed, theseinventions may be embodied in many different forms and should not beconstrued as limited to the embodiments set forth herein; rather, theseembodiments are provided so that this disclosure will satisfy applicablelegal requirements. Like numbers refer to like elements throughout.

Overview

As noted above, methods, apparatuses, and systems are described hereinthat provide solutions to the problems identified above, as well asothers. In one example embodiment, an eye tracking system is providedthat can efficiently toggle between bright and dark pupil identificationmethods. As shown in FIG. 1A, bright pupil identification comprises apupil identification method in which a light source is disposed in aphysically close proximity to a camera that detects reflection of thelight from a user's pupil. The reflected light creates an effect similarto the red-eye effect commonly experienced during regular photography.In contrast, dark pupil identification comprises a pupil identificationmethod in which a light source is disposed farther away from the camera,and while the reflected light from the farther-out light source is usedfor pupil identification, the underlying mechanism by which theidentification is performed is different. Accordingly, exampleembodiments thus enable efficient toggling between these distinct pupilidentification methods in order to provide an enhanced pupilidentification method, apparatus, and system.

In another example embodiment not mutually exclusive of the first, anapparatus is configured for pupil identification while also having aform factor and functionality similar to that of the smartphone ortablet devices currently utilized by consumers today. And in yet anotherexample embodiment not mutually exclusive of the first or second, anexisting smartphone or tablet device is configured to enable performanceof pupil identification and, in turn, provide eye tracking functionalitythat historically has required special-purpose equipment.

Greater detail is provided below regarding certain example embodimentscontemplated herein.

Device Architecture

Apparatuses of the present invention may be embodied by any of a varietyof devices. For example, the apparatus performing improved pupilidentification may include any of a variety of fixed terminals, such aserver, desktop, or kiosk, or it may comprise any of a variety of mobileterminals, such as a portable digital assistant (PDA), mobile telephone,smartphone, laptop computer, tablet computer, or in some embodiments, aperipheral device that connects to one or more fixed or mobileterminals. Example embodiments contemplated herein may have various formfactors and designs, but will nevertheless include at least thecomponents illustrated in FIG. 2 and described in connection withexample apparatus 200.

As illustrated in FIG. 2, the apparatus 200 may include a processor 202,a memory 204, input/output circuitry 206, and communications circuitry208. Moreover, apparatus 200 may include toggle circuitry 210, brightpupil identification circuitry 212, dark pupil identification circuitry214, eye tracking circuitry 216, and in some embodiments, sentimentanalysis circuitry 218. The apparatus 200 may be configured to executethe operations described below in connection with FIG. 5. Although thesecomponents 202-218 are described in some cases using functionallanguage, it should be understood that the particular implementationsnecessarily include the use of particular hardware. It should also beunderstood that certain of these components 202-218 may include similaror common hardware. For example, two sets of circuitry may both leverageuse of the same processor 202, memory 204, communications circuitry 208,or the like to perform their associated functions, such that duplicatehardware is not required for each set of circuitry. The use of the term“circuitry” as used herein with respect to components of the apparatustherefore includes particular hardware configured to perform thefunctions associated with respective circuitry described herein.

Of course, while the term “circuitry” should be understood broadly toinclude hardware, in some embodiments, the term “circuitry” may alsoinclude software for configuring the hardware. For example, although“circuitry” may include processing circuitry, storage media, networkinterfaces, input/output devices, and the like, other elements of theapparatus 200 may provide or supplement the functionality of particularcircuitry.

In some embodiments, the processor 202 (and/or co-processor or any otherprocessing circuitry assisting or otherwise associated with theprocessor) may be in communication with the memory 204 via a bus forpassing information among components of the apparatus. The memory 204may be non-transitory and may include, for example, one or more volatileand/or non-volatile memories. In other words, for example, the memorymay be an electronic storage device (e.g., a non-transitory computerreadable storage medium). The memory 204 may be configured to storeinformation, data, content, applications, instructions, or the like, forenabling the apparatus to carry out various functions in accordance withexample embodiments of the present invention.

The processor 202 may be embodied in a number of different ways and may,for example, include one or more processing devices configured toperform independently. Additionally or alternatively, the processor mayinclude one or more processors configured in tandem via a bus to enableindependent execution of instructions, pipelining, and/ormultithreading. The use of the term “processing circuitry” may beunderstood to include a single core processor, a multi-core processor,multiple processors internal to the apparatus, and/or remote or “cloud”processors.

In an example embodiment, the processor 202 may be configured to executeinstructions stored in the memory 204 or otherwise accessible to theprocessor 202. Alternatively or additionally, the processor 202 may beconfigured to execute hard-coded functionality. As such, whetherconfigured by hardware or by a combination of hardware with software,the processor 202 may represent an entity (e.g., physically embodied incircuitry) capable of performing operations according to an embodimentof the present invention while configured accordingly. Alternatively, asanother example, when the processor 202 is embodied as an executor ofsoftware instructions, the instructions may specifically configure theprocessor 202 to perform the algorithms and/or operations describedherein when the instructions are executed.

The apparatus 200 further includes input/output circuitry 206 that may,in turn, be in communication with processor 202 to provide output to theuser and to receive input from a user or another source. In this regard,the input/output circuitry may comprise a camera configured to captureimages and/or detect infrared light reflected from a user's eye. Theinput/output circuitry may further comprise a separate light sensor thatis configured to detect light levels in the environment and which may bedisposed in the apparatus 200 on the same surface as a camera. The lightsensor may in this regard be an infrared light sensor. Separately, theinput/output circuitry 206 may comprise a display that may bemanipulated by a mobile application. In some embodiments, theinput/output circuitry 206 may also include additional functionalitykeyboard, a mouse, a joystick, a touch screen, touch areas, soft keys, amicrophone, a speaker, or other input/output mechanisms. The processor202 and/or user interface circuitry comprising the processor 202 may beconfigured to control one or more functions of display through computerprogram instructions (e.g., software and/or firmware) stored on a memoryaccessible to the processor (e.g., memory 204, and/or the like), such asto cause blocks of the display to emit light at selected wavelengths.

The communications circuitry 208 may be any means such as a device orcircuitry embodied in either hardware or a combination of hardware andsoftware that is configured to receive and/or transmit data from/to anetwork and/or any other device, circuitry, or module in communicationwith the apparatus 200. In this regard, the communications circuitry 208may include, for example, a network interface for enablingcommunications with a wired or wireless communication network. Forexample, the communications circuitry 208 may include one or morenetwork interface cards, antennae, buses, switches, routers, modems, andsupporting hardware and/or software, or any other device suitable forenabling communications via a network. Additionally or alternatively,the communication interface may include the circuitry for interactingwith the antenna(s) to cause transmission of signals via the antenna(s)or to handle receipt of signals received via the antenna(s). Thesesignals may be transmitted by the apparatus 300 using any of a number ofwireless personal area network (PAN) technologies, such as Bluetooth®v1.0 through v3.0, Bluetooth Low Energy (BLE), infrared wireless (e.g.,IrDA), ultra-wideband (UWB), induction wireless transmission, or thelike. In addition, it should be understood that these signals may betransmitted using Wi-Fi, Near Field Communications (NFC), WorldwideInteroperability for Microwave Access (WiMAX) or other proximity-basedcommunications protocols.

Toggle circuitry 210 includes hardware components designed to determinewhether a measured ambient infrared light level exceeds a thresholdlevel. These hardware components may, for instance, utilize elements ofinput/output circuitry 206 to detect ambient infrared light levels, andmemory 204 to retrieve a stored value of the threshold level. Togglecircuitry 210 may utilize processing circuitry, such as the processor202, to perform the above operations, and may utilize memory 204 tostore collected information.

Bright pupil identification circuitry 212 includes hardware componentsdesigned to perform a bright pupil identification process in an instancein which the measured ambient infrared light level exceeds a thresholdlevel. These hardware components may, for instance, utilize elements ofinput/output circuitry 206 to detect ambient infrared light levels, andmemory 204 to retrieve a stored value of the threshold level. Brightpupil identification circuitry 212 may utilize processing circuitry,such as the processor 202, to perform its corresponding operations, andmay utilize memory 204 to store collected information.

Dark pupil identification circuitry 212 includes hardware componentsdesigned to perform a dark pupil identification process in an instancein which the measured ambient infrared light level does not exceed thethreshold level. These hardware components may, for instance, utilizeelements of input/output circuitry 206 to detect ambient infrared lightlevels, and memory 204 to retrieve a stored value of the thresholdlevel. Bright pupil identification circuitry 212 may utilize processingcircuitry, such as the processor 202, to perform its correspondingoperations, and may utilize memory 204 to store collected information.

Eye tracking circuitry 216 includes hardware components designed totrack eye movements of a user. These hardware components may, forinstance, utilize bright pupil identification circuitry 212 or darkpupil identification circuitry 214 to perform underlying pupilidentification facilitating the eye tracking functionality. Eye trackingcircuitry 216 may utilize processing circuitry, such as the processor202, to perform the above operations, and may utilize memory 204 tostore collected information.

Finally, sentiment analysis circuitry 218 includes hardware componentsdesigned to performing user sentiment analysis based on a historicalrecord of the user's eye movements captured. These hardware componentsmay, for instance, utilize elements eye tracking circuitry 216 tocapture the eye movements used for sentiment analysis. Sentimentanalysis circuitry 218 may utilize processing circuitry, such as theprocessor 202, to perform the above operations, and may utilize memory204 to store collected information.

It should also be appreciated that, in some embodiments, the togglecircuitry 210, bright pupil identification circuitry 212, dark pupilidentification circuitry 214, eye tracking circuitry 216, or sentimentanalysis circuitry 218 may include a separate processor, speciallyconfigured field programmable gate array (FPGA), or application specificinterface circuit (ASIC) to perform its corresponding functions.

In addition, computer program instructions and/or other type of code maybe loaded onto a computer, processor or other programmable apparatus'scircuitry to produce a machine, such that the computer, processor otherprogrammable circuitry that execute the code on the machine create themeans for implementing the various functions, including those describedin connection with the components of apparatus 200.

Turning now to FIG. 3, an example apparatus 300 is shown that may embodythe components described above in connection with FIG. 2. As shown,apparatus 300 comprises a mobile device, and may embody a smartphone ortablet device of a kind similar to those known today. However, unliketraditional smartphones or tablet devices, apparatus 300 includesadditional components enabling it to perform additional eye trackingfunctionality in addition to the varied existing uses of mobile devicesof this nature. In relevant part, apparatus 300 includes a camera 302and additionally includes an infrared light sensor 304 configured tomeasure a level of ambient infrared light radiating on the front face ofthe apparatus 300. In addition, the apparatus 300 include a firstillumination source 306 and a second illumination source 308. The firstillumination source 306 and the camera 302 are disposed on the same faceof the apparatus 300. And as can be seen from FIG. 3, the firstillumination source 306 is disposed in close proximity to the camera302. By virtue of its close proximity to the camera 302, the detectedreflection of light emitted from the first illumination source 306creates an effect similar to the red-eye effect often experienced inregular photography. In contrast, the second illumination source 308,although disposed on the same face of the apparatus 300 as the camera302, is disposed at a location on the face of the apparatus 300 that isphysically distant from the camera 302. Introducing this physical gapprovides a wide angle of reflection of any infrared light reflected fromthe second illumination source, to a user's eye, and to the camera 302.As one example, the typical person carries their smartphone about 40 cmaway from their face, so when the apparatus 300 comprises a smartphone,because the second illumination source 308 is at an opposite end of thefront face of the smartphone, there is a roughly 30 degree angle ofreflection of light reaching the camera 302.

FIG. 4 illustrates another example apparatus 400 that may embody thecomponents described above in connection with FIG. 2. In contrast toapparatus 300, however, apparatus 400 may comprise any of a wide rangeof apparatuses that exist in the market today, and does not describeexternal physical changes to those apparatuses. Instead, apparatus 400describes a similar physical device but which has had a fundamentalimprovement to its functioning provided by way of software unlocking newcapabilities heretofore nonexistent in the device. As shown in FIG. 3,the apparatus 400 illustrated in FIG. 4 includes a camera 402, and aninfrared light sensor 404 of a similar nature and kind as thosedescribed in connection with FIG. 3. But instead of physicalillumination sources 306 and 308, the first illumination source 406 ofapparatus 400 is a block of pixels nearby the camera 402. The apparatus400 causes the first illumination source 406 (i.e., the block of pixels)to emit light at a wavelength as close to the infrared spectrum aspossible based on the technical design of the screen of the apparatus400, which may vary based upon which device is loaded with the softwarefunctionality described herein. This light may comprise far-red,near-infrared, or even pure infrared light in some embodiments.Accordingly, apparatus 400 is capable of performing bright pupilidentification without a dedicated illumination source such as firstillumination source 306. In some embodiments, the components housedwithin the aperture of the camera 402 may include infrared emittingcapability, and embodiments of apparatus 400 with this capability mayutilize that infrared emitting capability as the first illuminationsource 406. In similar fashion, the second illumination source 408 ofapparatus 400 comprises a block of pixels located away from the cameraelement of the apparatus, as further shown in FIG. 4, thus obviating theneed for a dedicated illumination source such as second illuminationsource 308 that is provided in apparatus 300.

As described above and as will be appreciated based on this disclosure,embodiments of the present invention may be configured as systems,methods, mobile devices, and the like. Accordingly, embodiments maycomprise various means including entirely of hardware or any combinationof software with hardware. Furthermore, embodiments may take the form ofa computer program product comprising instructions stored on at leastone non-transitory computer-readable storage medium (e.g., computersoftware stored on a hardware device). Any suitable computer-readablestorage medium may be utilized including non-transitory hard disks,CD-ROMs, flash memory, optical storage devices, or magnetic storagedevices.

Having described specific components of example apparatuses 200, 300,and 400 in connection with FIGS. 2-4, an example procedure is describedbelow in connection with FIGS. 5 and 6 for improved pupil identificationusing both bright and/or dark pupil identification processes.

Example Operations for Enhanced Pupil Identification

Turning to FIG. 5, a flowchart is illustrated that contain operationsfor using an apparatus to perform pupil identification in a moreefficient and effective manner than has been provided by existingtechnologies. The operations illustrated in FIG. 5 may, for example, beperformed by an apparatus 300 or 400, as described above. And in thisregard, the apparatus 200 illustrates example components included ineither of apparatuses 300 or 400 in various embodiments, such that theoperations described herein may be performed by or through the use ofone or more of processor 202, memory 204, input/output circuitry 206,communications circuitry 208, toggle circuitry 210, bright pupilidentification circuitry 212, dark pupil identification circuitry 214,eye tracking circuitry 216, and/or sentiment analysis circuitry 218.

Operation 502, the first operation in the procedure illustrated in FIG.5, is optionally performed in some embodiments but not all. Whenperformed, the apparatus 300 includes means, such as processor 202,memory 204, input/output circuitry 206, communications circuitry 208,toggle circuitry 210, bright pupil identification circuitry 212, darkpupil identification circuitry 214, and/or the like, for calibrating theapparatus for the user to identify a threshold level above which brightpupil identification will be used, and at or below which dark pupilidentification will be used. This operation is optional becausecalibration is only one way of identifying an appropriate thresholdlevel to demarcate appropriate uses of bright and dark pupilidentification methods. When performed, this calibration operationtypically occurs prior to the remaining operations described below,although in other embodiments it may be performed sporadically as ameans of enhancing the beneficial effect of the toggling operationsdescribed herein by improving the selection of an appropriate thresholdlevel for toggling between pupil identification methods.

As noted above, calibration of the threshold level is designed toidentify the threshold level of ambient infrared light that maximizesoverall performance of pupil identification for a particular individual.In this regard, dark pupil identification is more effective than brightpupil identification given lower levels of ambient infrared light (e.g.,below about 5% of the ambient infrared light provided by sunlight at sealevel at the Sun's zenith and for a mean Earth-Sun separation), whilebright pupil identification is more effective than dark pupilidentification at higher levels of ambient infrared light (e.g., above20% of the ambient infrared light provided by sunlight at sea level atthe Sun's zenith and for a mean Earth-Sun separation). However, withinthese two ranges, the effectiveness of dark versus bright pupilidentification techniques can vary based on additional factors. Forinstance, dark pupil identification is generally a more accurate pupilidentification method for people of Asian or African origin, whilebright pupil identification is generally a more accurate pupilidentification method for people of Caucasian or Hispanic origin.Accordingly, the threshold value that maximizes effectiveness of pupilidentification in fluctuating light environments will be generally fallsomewhere within the 5% and 20% figures noted above.

Calibration of this threshold level prior to performance of theadditional operations illustrated in FIG. 5 thus provides a mechanism tooptimize the effectiveness of these operations. Calibration of thethreshold level may be performed using the apparatus 200 as follows.

First, the apparatus 200 may include means, such as an infrared lightsensor disposed on a surface of the apparatus, for measuring an ambientinfrared light level. The apparatus 200 may include means, such asinput/output circuitry 206, for instructing a user to look at a specificpoint on the apparatus. Subsequently, the apparatus 200 may includemeans, such as bright pupil identification circuitry 212 for performingbright pupil identification process using a first illumination source.With prior knowledge of the specific point on the apparatus at which theuser is looking, the apparatus 200 may include means, such as processor202 or the like, for determining an accuracy of the pupil identificationprocess performed using the bright pupil identification circuitry 212.Thereafter, the apparatus 200 may include means, such as dark pupilidentification circuitry 214 for performing pupil identification processusing a second illumination source. Of course, in some embodiments, darkpupil identification may be performed prior to performance of brightpupil identification. With prior knowledge of the specific point on theapparatus at which the user is looking, the apparatus 200 may includemeans, such as processor 202 or the like, for determining an accuracy ofthe pupil identification process performed using the dark pupilidentification circuitry 214. Finally, the apparatus 200 may includemeans, such as processor 202 or the like, for determining a relativeaccuracy of the bright pupil identification process and the dark pupilidentification process for the individual and given the ambient infraredlight level measured during the training period.

However, performing this calibration process a single time mayillustrate only that at a given level of ambient infrared light, eitherdark or bright pupil identification is more accurate for the individual.Accordingly, the calibration operation may be repeated a number of timesand in a number of environmental lighting conditions, to identify anideal inflection point above which bright pupil identification isgenerally more effective for the individual and below which dark pupilidentification is generally more effective for the individual. Becauseenvironmental lighting conditions are not controlled by the apparatus200, it may be the case that this calibration operation cannot berepeated at a single point in time. Accordingly, in some embodiments,the apparatus 200 may include means, such as computer instructionsstored on memory 204, which cause the apparatus 200 to prompt the userto perform a calibration operation at various times. For instance, insome embodiments, the apparatus 200 may simply prompt the userperiodically to perform additional calibration operations. As anotherexample, however, in some embodiments the infrared light sensor of theapparatus may be instructed by the apparatus 200 to periodically measurean ambient infrared light level, and upon measuring an ambient infraredlight level that diverges from ambient light levels used in priorcalibration operations, the apparatus 200 may prompt the user to performanother calibration operation to enhance the ability of the apparatus200 to identify an optimal threshold level for pupil identification.

If operation 502 is performed in a given circumstance, the procedureillustrated in FIG. 5 then advances to operation 504, although ifoperation 502 is not performed, then the procedure may simply begin atoperation 504. Either way, at operation 504, the apparatus 200 includesmeans, such as input/output circuitry 206 or the like, for measuring anambient infrared light level. In this regard, input/output circuitry 206may most commonly comprise a light sensor. The light sensor may be aninfrared light sensor as known in the art, and may, for instance,comprise one or more passive infrared sensors (e.g., pyroelectricdevices). There may be a single light sensing component, or there may bemultiple light sensing components disposed at various locations aroundthe apparatus 200 that provides more information regarding the ambientinfrared radiation received by the apparatus 200.

Turning next to operation 506, the apparatus 200 includes means, such astoggle circuitry 210 or the like, for determining whether to perform adark pupil identification process or a bright pupil identificationprocess. In this regard, the toggle circuitry 210 is configured to makethis determination based on the measured ambient infrared light level.For instance, the bright pupil identification process may be associatedwith a first range of ambient infrared radiation, while the dark pupilidentification process may be associated with a second range of ambientinfrared radiation. Accordingly, in an instance in which the measureambient infrared light level falls within the first range, the procedureadvances to operation 508 for performance of bright pupilidentification. However, in an instance in which the measured ambientinfrared light level falls within the second range, the procedureadvances to operation 510 for performance of dark pupil identification.

In some embodiments, the first range and the second range are separatedby a predetermined threshold level, such that a level of ambientinfrared radiation below or equal to the predetermined threshold levelfalls within the first range, while a level of ambient infraredradiation over the predetermined threshold level falls within the secondrange. This threshold level may have been predetermined using acalibration procedure described in connection with operation 502.Alternatively, the threshold level may be preset by a manufacturer at adefault level within the range of 5% to 20% of the ambient infraredlight provided by sunlight at sea level at the Sun's zenith and for amean Earth-Sun separation. In still further embodiments, the thresholdlevel may be manually selected and adjusted by a user via a userinterface presented by the apparatus 200. For instance, the userinterface 600 illustrated in FIG. 6 may be used to determine anethnicity of the user, and upon selection of appropriate radio buttonsand selection of a “Submit” button 602, the received ethnicityidentification information selected by a user may be used by theapparatus 200 to adjust the default threshold value up or down based ongeneralized knowledge of the efficacy of dark and bright pupilidentification methods on people of different ethnicities.

In some embodiments, the first range does not directly abut the secondrange, and in such embodiments, there will be some intermediate levelsof ambient infrared radiation that would not fall within either thefirst range or the second range (indicating that such levels of ambientinfrared radiation are not suitable for either the dark pupilidentification process or the bright pupil identification process). Insuch embodiments, rather than a single threshold level, there may be afirst threshold level below which the bright pupil identificationprocess is not suitable, and a second threshold level above which thedark pupil identification process is not suitable. The determination ofthese first and second threshold levels may be made during a calibrationprocedure similar to that described above, or may be preset, manuallyselected, and/or manually adjusted, in similar fashion as describedabove for situations where only a single threshold level is present.Finally, there may be a level of ambient infrared radiation above whicheven the bright pupil identification process would not be accurate, andin such embodiments, the first range may be bounded on the high side bya third threshold level above which the bright pupil identificationprocess is not accurate. In embodiments where certain levels of ambientinfrared light do not fall within either the first range or the secondrange, there will be levels of measured ambient infrared light for whichthe toggle circuitry 210 may thus determine that the apparatus 200 isunable to perform pupil identification by either mechanism, because themeasured ambient infrared light level falls outside of either the firstrange or the second range. In such circumstances, the procedure advancesto (and ends) at operation 512.

Turning next to operations 508 and 510, it will be understood that thefunctionality described in connection with those operations may bereferred to generally herein as being performed by pupil identificationcircuitry of the apparatus 200. Discussion of pupil identificationcircuitry of the apparatus 200 at this level of breadth is meant toillustrate that various embodiments described herein are configured forperforming pupil identification on the user based on the determinationmade by toggle circuitry 210, in contrast to exiting techniques forpupil identification which are not designed to operate in response todecision-making premised on measured ambient infrared light levels. At amore granular level, however, the pupil identification circuitry ofapparatus 200 may comprise bright pupil identification circuitry 212 anddark pupil identification circuitry 214.

As shown at operation 508, the apparatus 200 includes means, such asbright pupil identification circuitry 212, for performing a bright pupilidentification process in an instance in which the measured ambientinfrared light level falls within the first range identified above. Andas shown by operation 510, the apparatus 200 includes means, such asdark pupil identification circuitry 214, for performing a dark pupilidentification process in an instance in which the measured ambientinfrared light level falls within the second range identified above.

For the bright pupil identification process, the bright pupilidentification circuitry 212 causes the first illumination source toemit light, and causes input/output circuitry 206 (which uses a camera,in most cases) to detect received infrared light. As noted previously indiscussing apparatus 300, the first illumination source and the camera(or other component of the input/output circuitry 206 designed fordetecting infrared light) are disposed on the same general face of theapparatus (which for a smartphone, would be the front of thesmartphone), so that when the face of the apparatus is pointed towards auser's eye, the input/output circuitry 206 will receive infrared lightreflected by the user's eye. The first illumination source, in thisregard, is disposed in close proximity to the camera or other componentof the input/output circuitry 206 designed for detecting reflectedinfrared light. By virtue of its close proximity to the camera (or othercomponent), the detected reflection of the emitted light creates aneffect similar to the red-eye effect often experienced in regularphotography. The bright pupil identification circuitry 212 deduces aposition, size, and orientation of the user's pupil by virtue of itsdifferent reflecting properties from other parts of the user' eye andface.

For the dark pupil identification process, the dark pupil identificationcircuitry 214 causes the second illumination source to emit light, andcauses input/output circuitry 206 (which uses a camera, in most cases)to detect received infrared light. The second illumination source andthe camera (or other component of the input/output circuitry 206designed for detecting infrared light) are disposed on the same generalface of the apparatus (which for a smartphone, would be the front of thesmartphone), so that when the face of the apparatus is pointed towards auser's eye, the input/output circuitry 206 will receive infrared lightreflected by the user's eye. But unlike the first illumination source,the second illumination source is disposed at a location on the face ofthe apparatus that is physically distant from the camera or othercomponent of the input/output circuitry 206 designed for detectingreflected infrared light. Introducing this physical gap provides a wideangle of reflection of any infrared light reflected by the eye. As oneexample, the typical person carries their smartphone about 40 cm awayfrom their face, so when the second illumination source is at anopposite end of the front face of the smartphone, there is a roughly 30degree angle of reflection of light reaching the camera or othercomponent of the input/output circuitry 206 designed for detecting thereflected infrared light. Moreover, even if one holds the phone at twicethis distance, the resulting 15 degree angle of reflection remainsadequate for dark pupil identification. Accordingly, in contrast to abright pupil identification process, the dark pupil identificationcircuitry 214 relies on an analysis of different characteristics of thereflected light to deduce a position, size, and orientation of theuser's pupil by virtue of its different reflecting properties from otherparts of the user' eye and face, as will be understood by those familiarwith existing eye tracking systems.

As described above in connection with FIGS. 3 and 4, the first andsecond illumination sources can take multiple forms. As shown in FIG. 3,the first illumination source may be a dedicated infrared illuminator.But as shown in FIG. 4, the first illumination source may instead be ablock of pixels nearby the camera element of a smartphone or tabledevice. In such embodiments, the apparatus 200 may cause the firstillumination source (i.e., the block of pixels) to emit light at awavelength as close to the infrared spectrum as possible based on thetechnical design of the screen of the apparatus. This light may comprisefar-red or near-infra-red, or even pure infrared light, depending on thephysical capabilities of the display components of the apparatus itself.Accordingly, a mobile device such as a smartphone or a tablet device maybe capable of performing bright pupil identification by including withinit an infrared light sensor in close physical proximity to the cameraaperture, or an existing device may be rendered capable of performingbright pupil identification by using a nearby blocks of pixels insteadof a separate dedicated illumination element. Finally, in someembodiments, the components housed within the aperture of manysmartphone and tablet devices may include infrared emitting capability,in which case this capability may separately be harnessed to operate asthe first illumination source in some embodiments, without requiring theexistence of a separate dedicated illumination element. In similarfashion, the second illumination source may be a dedicated infraredilluminator as shown in FIG. 3, or it may be a block of pixels locatedaway from the camera element of the apparatus, as shown in FIG. 4.

Regardless of whether a dark or bright pupil identification process isused in any given circumstance and regardless of the type ofillumination source used to perform the corresponding pupilidentification operation, the results of pupil identification fromoperations 508 or 510 may subsequently be used in operation 514.

As illustrated at operation 514, the apparatus 300 includes means, suchas eye tracking circuitry 216, or the like, for tracking eye movementsof a user. To do this, eye tracking circuitry 216 may extrapolate a gazedirection and distance of a user's eye from the known position, size,and orientation of the pupil in a manner that will be understood bythose familiar with existing eye tracking technologies. It will beunderstood that operation 514 will frequently follow performance ofpupil identification in operations 508 or 510, but it is not necessaryin every embodiment contemplated here.

And finally, in circumstances where optional operation 516 is invoked,the apparatus 300 includes means, such as sentiment analysis circuitry218, or the like, for performing user sentiment analysis based on ahistorical record of the user's eye movements captured in operation 514.It will be understood that operation 516 is optional, because eyetracking may be used for many purposes, and only some of them will befocused on the sentiment of the user.

As described herein, example embodiments thus provide many benefits toeye tracking systems that have heretofore gone unrealized. Exampleembodiments described herein provide an eye tracking system configuredto efficiently toggle between pupil identification methods in anefficient yet accurate manner. In addition, some example embodimentsutilize an apparatus configured for pupil identification that has a formfactor and functionality similar to that of the smartphone or tabletdevices currently utilized by consumers today. And finally, some exampleembodiments contemplate the modification of existing smartphone ortablet devices to perform pupil identification that has heretofore beenimpossible without special-purpose equipment.

FIG. 5 thus illustrates a flowchart describing the operation ofapparatuses, methods, and computer program products according to exampleembodiments contemplated herein. It will be understood that each blockof the flowchart, and combinations of blocks in the flowchart, may beimplemented by various means, such as hardware, firmware, processor,circuitry, and/or other devices associated with execution of softwareincluding one or more computer program instructions. For example, one ormore of the procedures described above may be embodied by computerprogram instructions. In this regard, the computer program instructionswhich embody the procedures described above may be stored by a memory204 of an apparatus 200 and executed by a processor 202 of the apparatus200. As will be appreciated, any such computer program instructions maybe loaded onto a computer or other programmable apparatus (e.g.,hardware) to produce a machine, such that the resulting computer orother programmable apparatus implements the functions specified in theflowchart blocks. These computer program instructions may also be storedin a computer-readable memory that may direct a computer or otherprogrammable apparatus to function in a particular manner, such that theinstructions stored in the computer-readable memory produce an articleof manufacture, the execution of which implements the functionsspecified in the flowchart blocks. The computer program instructions mayalso be loaded onto a computer or other programmable apparatus to causea series of operations to be performed on the computer or otherprogrammable apparatus to produce a computer-implemented process suchthat the instructions executed on the computer or other programmableapparatus provide operations for implementing the functions specified inthe flowchart blocks.

The flowchart blocks support combinations of means for performing thespecified functions and combinations of operations for performing thespecified functions. It will be understood that one or more blocks ofthe flowcharts, and combinations of blocks in the flowcharts, can beimplemented by special purpose hardware-based computer systems whichperform the specified functions, or combinations of special purposehardware with computer instructions.

Conclusion

Many modifications and other embodiments of the inventions set forthherein will come to mind to one skilled in the art to which theseinventions pertain having the benefit of the teachings presented in theforegoing descriptions and the associated drawings. Therefore, it is tobe understood that the inventions are not to be limited to the specificembodiments disclosed and that modifications and other embodiments areintended to be included within the scope of the appended claims.Moreover, although the foregoing descriptions and the associateddrawings describe example embodiments in the context of certain examplecombinations of elements and/or functions, it should be appreciated thatdifferent combinations of elements and/or functions may be provided byalternative embodiments without departing from the scope of the appendedclaims. In this regard, for example, different combinations of elementsand/or functions than those explicitly described above are alsocontemplated as may be set forth in some of the appended claims.Although specific terms are employed herein, they are used in a genericand descriptive sense only and not for purposes of limitation.

What is claimed is:
 1. An apparatus for performing pupil identification,the apparatus comprising: a light sensor configured to measure anambient infrared light level; toggling circuitry configured todetermine, based on the measured ambient infrared light level fallingwithin a first range of ambient infrared radiation corresponding to abright pupil identification process or falling within a second range ofambient infrared radiation corresponding to a dark pupil identificationprocess, whether to perform pupil identification using only one ofeither the dark pupil identification process or the bright pupilidentification process; and pupil identification circuitry configured toperform pupil identification on a user using the dark pupilidentification process upon the determination that the measured ambientinfrared light level falls within the second range of ambient infraredradiation; and to perform pupil identification on the user using thebright identification process upon the determination that the measuredambient infrared light level falls within the first range of ambientinfrared radiation.
 2. The apparatus of claim 1, wherein the pupilidentification circuitry comprises dark pupil identification circuitryand bright pupil identification circuitry, wherein the bright pupilidentification circuitry is configured to perform the bright pupilidentification process using a first illumination source of theapparatus, and wherein the dark pupil identification circuitry isconfigured to perform the dark pupil identification process using asecond illumination source of the apparatus.
 3. The apparatus of claim2, wherein the first illumination source or the second illuminationsource comprises an infrared illuminator.
 4. The apparatus of claim 2,wherein the first illumination source or the second illumination sourcecomprises a block of pixels on a face of the apparatus.
 5. The apparatusof claim 2, wherein the first range of ambient infrared radiationcomprises any amount of ambient infrared radiation above a thresholdlevel, and wherein the second range of ambient infrared radiationcomprises any amount of ambient infrared radiation at or below thethreshold level.
 6. The apparatus of claim 5, wherein the thresholdlevel comprises a particular level of ambient infrared light within arange of 5% to 20% of a level of radiant infrared light produced bysunlight at the Sun's zenith and for a mean Earth-Sun separation.
 7. Theapparatus of claim 1, further comprising: eye tracking circuitryconfigured to track one or more eyes of the particular user based onperforming the pupil identification; and sentiment analysis circuitryconfigured to analyze a sentiment of the user based on the tracking ofthe one or more eyes of the user.
 8. The apparatus of claim 1, whereinthe apparatus comprises a mobile device.
 9. The apparatus of claim 8,wherein the apparatus comprises a smartphone or a tablet device.
 10. Amethod for using an apparatus to perform pupil identification on a user,the method comprising: measuring, by a light sensor of the apparatus, anambient infrared light level; determining, by toggling circuitry of theapparatus and based on the measured ambient infrared light level fallingwithin a first range of ambient infrared radiation corresponding to abright pupil identification process or falling within a second range ofambient infrared radiation corresponding to a dark pupil identificationprocess, whether to perform pupil identification using only one ofeither the dark pupil identification process or the bright pupilidentification process; performing pupil identification on the userusing the dark pupil identification process upon the determination thatthe measured ambient infrared light level falls within the second rangeof ambient infrared radiation, and performing pupil identification onthe user using the bright pupil identification process upon thedetermination that the measured ambient infrared light level fallswithin the first range of ambient infrared radiation.
 11. The method ofclaim 10, wherein performing pupil identification on the user comprises:in an instance in which the measured ambient infrared light level fallswithin the first range of ambient infrared radiation, performing, bybright pupil identification circuitry of the apparatus, the bright pupilidentification process using a first illumination source of theapparatus; and in an instance in which the measured ambient infraredlight level falls within the second range of ambient infrared radiation,performing, by dark pupil identification circuitry of the apparatus, thedark pupil identification process using a second illumination source.12. The method of claim 11, wherein the first illumination source or thesecond illumination source comprises an infrared illuminator.
 13. Themethod of claim 11, wherein the first illumination source or the secondillumination source comprises a block of pixels on a face of theapparatus.
 14. The method of claim 10, wherein the first range ofambient infrared radiation comprises any amount of ambient infraredradiation above a threshold level, and wherein the second range ofambient infrared radiation comprises any amount of ambient infraredradiation at or below the threshold level.
 15. The method of claim 14,wherein the threshold level comprises a particular level of ambientinfrared light within a range of 5% to 20% of a level of radiantinfrared light from sunlight at the sun's zenith and for a meanEarth-sun separation.
 16. The method of claim 14, further comprising:receiving a default threshold value; receiving an indication of anethnicity of a user; and determining the threshold level by adjustingthe default threshold value based on the indication of the ethnicity ofthe user.
 17. The method of claim 14, further comprising: prior tomeasuring the ambient infrared light level, calibrating the apparatusfor the user to identify the threshold level.
 18. The method of claim10, further comprising: tracking one or more eyes of the user based onperforming the pupil identification.
 19. The method of claim 18, furthercomprising: performing user sentiment analysis based on the tracking ofthe one or more eyes of the user.
 20. A non-transitory computer-readablestorage medium for using an apparatus to perform pupil identification,the non-transitory computer-readable storage medium storing instructionsthat, when executed, cause the apparatus to: measure, using a lightsensor, an ambient infrared light level; determine, using togglingcircuitry and based on the measured ambient infrared light level fallingwithin a first range of ambient infrared radiation corresponding to abright pupil identification process or falling within a second range ofambient infrared radiation corresponding to a dark pupil identificationprocess, whether to perform pupil identification using only one ofeither the dark pupil identification process or the bright pupilidentification process; and perform, using pupil identificationcircuitry, pupil identification on a user using either the dark pupilidentification process upon the determination that the measured ambientinfrared light level falls within the second range of ambient infraredradiation; and to perform pupil identification on the user using thebright identification process upon the determination that the measuredambient infrared light level falls within the first range of ambientinfrared radiation.